Red-eye filter method and apparatus

ABSTRACT

A digital image acquisition system having no photographic film, such as a digital camera, has a flash unit for providing illumination during image capture and a red-eye filter for detecting a region within a captured image indicative of a red-eye phenomenon, the detection being based upon a comparison of the captured image and a reference image of nominally the same scene taken without flash. In the embodiment the reference image is a preview image of lower pixel resolution than the captured image, the filter matching the pixel resolutions of the captured and reference images by up-sampling the preview image and/or sub-sampling the captured image. The filter also aligns at least portions of the captured image and reference image prior to comparison to allow for, e.g. movement in the subject.

PRIORITY AND RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/119,614, filed May 13, 2008, published as US2008/0211937, which is acontinuation of Ser. No. 10/919,226, filed Aug. 16, 2004, published asUS2005/0041121, which is a continuation-in-part (CIP) application whichclaims the benefit of priority to U.S. patent application Ser. No.10/772,767, filed Feb. 4, 2004, now U.S. Pat. No. 7,352,394 issued Apr.1, 2008, which is a CIP that claims the benefit of priority to U.S.patent application Ser. No. 10/635,862, published as US2004/0223063which is a CIP that claims the benefit of priority to U.S. patentapplication Ser. No. 10/170,511, filed Jun. 12, 2002, now U.S. Pat. No.7,042,505 issued May 9, 2006, which is a continuation of U.S. patentapplication Ser. No. 08/947,603, filed Oct. 9, 1997, now U.S. Pat. No.6,407,777, issued Jun. 18, 2002, which are each hereby incorporated byreference. This application is also related to U.S. patent applicationsNo. 10/635,918, filed Aug. 5, 2003, published as US2005/0031224,10/773,092, filed Feb. 4, 2004, published as US2005/0140801, 11/379,346filed Apr. 19, 2006, 11/690,834 filed Mar. 25, 2007, published asUS2007/0263104, 11/769,206 filed Jun. 27, 2007 published asUS2008/0002060, 12/035,416 filed Feb. 21, 2008, PCT application no.PCT/EP2005/005907 filed Jun. 1, 2005 published as WO2006/018056.

FIELD OF THE INVENTION

The invention relates generally to the area of digital photography, andmore specifically to filtering “red-eye” artefacts from a flash-induceddigital camera image.

BACKGROUND OF THE INVENTION

“Red-eye” is a phenomenon in flash photography where a flash isreflected within a subject's eye and appears in a photograph as a reddot where the black pupil of the subject's eye would normally appear.The unnatural glowing red of an eye is due to internal reflections fromthe vascular membrane behind the retina, which is rich in blood vessels.This objectionable phenomenon is well understood to be caused in part bya small angle between the flash of the camera and the lens of thecamera. This angle has decreased with the miniaturization of cameraswith integral flash capabilities. Additional contributors include therelative closeness of the subject to the camera and ambient lightlevels.

The red-eye phenomenon can be minimized by causing the iris to reducethe opening of the pupil. This is typically done with a “pre-flash”, aflash or illumination of light shortly before a flash photograph istaken. This causes the iris to close. Unfortunately, the pre-flash is anobjectionable 0.2 to 0.6 seconds prior to the flash photograph. Thisdelay is readily discernible and easily within the reaction time of ahuman subject. Consequently the subject may believe the pre-flash is theactual photograph and be in a less than desirable position at the timeof the actual photograph. Alternately, the subject must be informed ofthe pre-flash, typically loosing any spontaneity of the subject capturedin the photograph.

Those familiar with the art have developed complex analysis processesoperating within a camera prior to invoking a pre-flash. Variousconditions are monitored prior to the photograph before the pre-flash isgenerated; the conditions include the ambient light level and thedistance of the subject from the camera. Such a system is described inU.S. Pat. No. 5,070,355, which is hereby incorporated by reference.Although that invention minimizes the occurrences where a pre-flash isused, it does not eliminate the need for a pre-flash.

Digital cameras are becoming more popular and smaller in size. Digitalcameras have several advantages over film cameras. Digital cameraseliminate the need for film as the image is digitally captured andstored in a memory array for display on a display screen on the cameraitself. This allows photographs to be viewed and enjoyed virtuallyinstantaneously as opposed to waiting for film processing. Furthermore,the digitally captured image may be downloaded to another display devicesuch as a personal computer or color printer for further enhancedviewing. Digital cameras include microprocessors for image processingand compression and camera systems control. It is possible to exploitthe computation capabilities of such microprocessors for performingoperations to improve the red-eye detection and elimination. Thus, whatis needed is a method of better tools for eliminating red-eye phenomenonwithin, for example, a digital camera having a flash unit without thedistraction of a pre-flash.

U.S. Patent Application 2002/0150306 (Baron), which is herebyincorporated by reference, describes a method for the removal of flashartefacts by capturing two digital images of a subject, one with flashand one without flash, and subtracting one image from the other toprovide an artefact image which is then thresholded and subtracted fromthe flash image. However, the technique is directed to flash artefactsin general, and not specifically to red-eye removal. There is no attemptto identify red-eye regions as compared to any other flash-inducedartefacts. Indeed, there is no attempt to identify particular regions atall, since the technique is simply one of subtraction and thresholding.

BRIEF SUMMARY OF THE INVENTION

A system in accordance with the present invention there is provided adigital image acquisition system having no photographic film, comprisinga portable apparatus for capturing digital images, a flash unit forproviding illumination during image capture, and a red-eye filter fordetecting a region within a captured image indicative of a red-eyephenomenon, said detection being based upon a comparison of saidcaptured image and at least one reference eye color characteristic.

According to one embodiment of the invention, the at least one referenceeye color characteristic includes a reference image of nominally thesame scene taken without flash.

According to this embodiment, the reference image may be a preview imageof lower pixel resolution than the captured image, the filter mayinclude programming code for matching the pixel resolutions of thecaptured and reference images by up-sampling the preview image and/orsub-sampling the captured image.

To allow for inadvertent movement in the subject between taking the twoimages, preferably the filter may further include programming code foraligning at least portions of the captured image and reference imageprior to said comparison.

In the embodiment the filter detects said region indicative of a red-eyephenomenon by identifying a region in the captured image at least havinga color indicative of a red-eye phenomenon and comparing said identifiedregion with the corresponding region in the reference image. The filtermay further designate the region as indicative of a red-eye phenomenonif said corresponding region does not have a color indicative of ared-eye phenomenon. The decision as to whether a region has a colorindicative of a red-eye phenomenon may be determined on a statisticalbasis as a global operation on the entire region.

According to a further embodiment, the filter includes a shape analyserso that the filter is configured to identify a region in the capturedimage having both a shape and color indicative of a red-eye phenomenonfor subsequent comparison with a reference eye shape characteristic andthe reference eye color characteristic, respectively.

In another embodiment, a pixel modifier is included for modifying thecolor of the pixels within a region indicative of a red-eye phenomenon.

A method in accordance with the present invention is also provided fordetecting a red-eye phenomenon within a digital image acquired by adigital image acquisition device having no photographic film. The deviceincludes a portable apparatus for capturing digital images, and a flashunit for providing illumination during image capture. The methodincludes identifying a region within a captured image indicative of ared-eye phenomenon including comparing the captured image and at leastone reference eye color characteristic.

According to one embodiment, the at least one reference eye colorcharacteristic includes an eye color characteristic of a reference imageof nominally the same scene taken without flash.

The reference image may be a preview image of lower pixel resolutionthan the captured image. The method may further include matching thepixel resolutions of the captured and reference images includingup-sampling the preview image or sub-sampling the captured image, or acombination thereof.

The method may further include aligning at least portions of thecaptured image and reference image prior to said comparison.

The method may further include analysing a shape so that the identifyingcomprises identifying a region in the captured image having both a shapeand color indicative of a red-eye phenomenon for subsequent comparisonwith the corresponding region in the reference image. A shape may beanalysed to determine subsequent to the comparison whether a regiondesignated as indicative of a red-eye phenomenon has a shape indicativeof a red-eye phenomenon.

The method may also include detecting a region indicative of a red-eyephenomenon by identifying a region in the captured image at least havinga color indicative of a red-eye phenomenon and comparing the identifiedregion with the corresponding region in the reference image, anddesignating the region as indicative of a red-eye phenomenon if thecorresponding region does not have a color indicative of a red-eyephenomenon.

The method may also include deciding whether a region has a colorindicative of a red-eye phenomenon by determining on a statistical basisas a global operation on the entire region.

The method may also include modifying the color of the pixels within aregion indicative of a red-eye phenomenon.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a camera apparatus operating in accordancewith an embodiment of the present invention.

FIG. 2 illustrates the workflow of the initial stage of a red-eye filterusing preview data according to the embodiment.

FIGS. 3-a to 3-d illustrates the pixelation process of an image indifferent resolutions.

FIG. 3-e is a enlargement of a hypothetical digitization of an eye in animage.

FIG. 4. illustrates the pixel differences between a red-eye image and anon red-eye image.

FIGS. 5-a to 5-d and 6-a and 6-b illustrate the detailed workflow of thered-eye filter according to the embodiment, and alternatives.

DESCRIPTION OF A PREFERRED EMBODIMENT

FIG. 1 shows a block diagram of a image acquisition system such as adigital camera apparatus operating in accordance with the presentinvention. The digital acquisition device, also generically referred toin this application as a camera 20, includes a processor 120. It can beappreciated that many of the processes implemented in the digital cameramay be implemented in or controlled by software operating in amicroprocessor (□Proc), central processing unit (CPU), controller,digital signal processor (DSP) and/or an application specific integratedcircuit (ASIC), collectively depicted as block 120 and termed as“processor”. Generically, all user interface and control of peripheralcomponents such as buttons and display is controlled by a □-controller122. The processor 120, in response to a user input at 122, such as halfpressing a shutter button (pre-capture mode 32), initiates and controlsthe digital photographic process. Ambient light exposure is determinedusing light sensor 40 in order to automatically determine if a flash isto be used. The distance to the subject is determined using focusingmeans 50 which also focuses the image on image capture means 60. If aflash is to be used, processor 120 causes the flash means 70 to generatea photographic flash in substantial coincidence with the recording ofthe image by image capture means 60 upon full depression of the shutterbutton. The image capture means 60 digitally records the image in color.The image capture means is known to those familiar with the art and mayinclude a CCD (charge coupled device) or CMOS to facilitate digitalrecording. The flash may be selectively generated either in response tothe light sensor 40 or a manual input 72 from the user of the camera.The image recorded by image capture means 60 is stored in image storemeans 80 which may comprise computer memory such a dynamic random accessmemory or a non-volatile memory. The camera is equipped with a display100, such as an LCD, for preview and post-view of images. In the case ofpreview images, which are generated in the pre-capture mode 32, thedisplay 100 can assist the user in composing the image, as well as beingused to determine focusing and exposure. In case of postview, the imagedisplay can assist the user in viewing suspected red-eye regions and tomanually decide if the region should be corrected or not after viewingit. A temporary storage space 82 is used to store one or plurality ofthe preview images and be part of the image store means 80 or a separatecomponent. The preview image is usually generated by the same imagecapture means 60, and for speed and memory efficiency reasons may begenerated by subsampling the image 124 using software which can be partof the general processor 120 or dedicated hardware, before displaying100 or storing 82 the preview image. Depending on the settings of thishardware subsystem, the pre-acquisition image processing may satisfysome predetermined criteria prior to storing the preview image. Suchcriteria may be chronological—such as save images every 0.5 seconds;more sophisticated criteria may be analysis of the image for changes, orthe detection of faces in the image. A straightforward preferredembodiment is to constantly replace the previous saved preview imagewith a new captured preview image during the pre-capture mode 32, untilthe final full resolution image is captured by full depression of theshutter button.

The red-eye filter 90 can be integral to the camera 20 or part of anexternal processing device 10 such as a desktop computer, a hand helddevice, a cell phone handset or a server. In this embodiment, the filterreceives the captured image from the full resolution image storage 80 aswell as one or a plurality of preview images from the temporary storage82. The filter 90 analyzes the stored image for characteristics ofred-eye and, if found, modifies the image and removes the red-eyephenomenon from the image as will be describe in more detail. Thered-eye filter includes a pixel locator 92 for locating pixels having acolor indicative of red-eye; a shape analyzer 94 for determining if agrouping of at least a portion of the pixels located by the pixellocator comprise a shape indicative of red-eye; an falsing analyzer 96for processing the image around the grouping for details indicative ofan image of an eye; and a pixel modifier 98 for modifying the color ofpixels within the grouping. The modified image may be either displayedon image display 100, saved on a persistent storage 112 which can beinternal or a removable storage such as CF card, SD card or the like, ordownloaded to another device, such as a personal computer, server orprinter via image output means 110 which can be tethered or wireless.

In a further embodiment where the red-eye filter 90 is located on anexternal application in a separate device, 10, such as a desktopcomputer, the final captured image stored in block 80 along with arepresentation of the preview image as temporarily stored in 82, may bestored prior to modification on the storage device 112, or transferredtogether via the image output means 110 onto the external device 10,later to be processed by the red-eye filter 90.

FIG. 2 details the initial stage of the workflow of this embodiment. Itwill be understood both this initial stage as well as the subsequentred-eye correction stage (See FIGS. 5 a to 5 d) will typically beperformed by software in the camera and/or separate device 10. A previewimage (normally of lesser resolution than the final image) is generatedwhile the camera is in the pre-capture mode 32 such as when the userhalf presses the shutter button. While in this mode, shown in FIG. 2 asthe preview mode 210, the camera constantly captures the preview images220. The capture interval is usually semi-real time which meansfractions of a tenth of a second or less. The camera saves each newpreview image if it satisfies some test criteria, 122. If not, thecamera continues, 211, to capture the next preview image without savingthe previous one. The process will continue until the final fullresolution image is acquired 280 and saved 282 by fully depressing theshutter button.

In a simple embodiment, if the test criteria are met, or if no testcriteria exist, the system will constantly replace the previous savedpreview image with the new preview image, 230. Alternatively, wheremultiple preview images can be saved, 240, the new image will be placedon a chronological FIFO stack, namely First In First Out, where thesystem continuously captures and saves new preview images 244 while eachtime clearing the oldest image 242 from the stack, until the user takesthe final picture. The reason for storing multiple preview images isbased on the fact that the last image, or any single image, may not bethe best reference image for comparison with the final full resolutionimage in the red-eye correction process. By storing multiple images, abetter reference image can be achieved, and a closer alignment betweenthe preview and the final captured image can be achieved. This conceptwill be further discussed in FIGS. 5 a to 5 c, in the alignment stage540. Other reasons for capturing multiple images are that a single imagemay be blurred due to motion, the subject had their eyes closed, theexposure was not set, etc. In a yet alternative embodiment, the multipleimages may assist in creating a single higher quality reference image;either higher resolution or by taking different portions of differentregions from the multiple images. This concept of sub-pixel resolutionmay be combined with the upsampling process as described in FIGS. 5 a to5 c, block 534.

The test criteria 222 may involve the actual analysis of the previewimage content before deciding whether the new preview image shouldreplace a previously saved image. Such criteria may be based on imageanalysis such as the existence of faces in the image, detection of eyesor metadata analysis such as the exposure condition, whether a flash isgoing to happen, the distance to the subjects, etc.

As part of the red-eye filter 90 the full resolution image 292 and thepreview image or images 294 will be loaded into working memory, 292 and294, unless they are already in memory in which they will just beaccessed through a pointer.

Referring to FIGS. 3 a-3 e, the digitization process in variousresolutions is explained and depicted. FIG. 3 a illustrates the gridlike nature of a sensor as illustrated in FIG. 1, block 60. The sensorcomprises multiple cells 302 which determine the camera resolution. Forexample a sensor of 2000×3000 cells will be a 6 Million pixel sensor (itwill be understood that each cell in fact comprises a plurality ofindividual sensor elements sensitive to different colors, e.g. RGB orRGBG, to create each colored image pixel).

FIG. 3-b depicts the optical projection of a face 310 onto the sensor60. Each of the cells 302 records the average light information itreceives for the image. This is the process of digitization andquantization.

The degree of details is determined by the resolution of the sensor asdepicted in FIG. 3-c. In this illustration a much smaller sensor isused, and in this case the same face 310 is digitized into a smallernumber of pixels, or alternatively subsampled from the full resolutionsensor data into a smaller number of pixel representations.

FIG. 3-d describes the inverse process where the subsampled image ofFIG. 3-c is upsampled to the same size as the original. When comparingthe resolution, naturally the some of the details are lost in thisprocess. For illustrative example, while in FIG. 3-b the face wasroughly 25×25=625 pixels, in FIG. 3-d the face is made of only 5×5=25pixels.

Of course, the above are only for illustration purposes. In practice,due to the larger resolution of the sensors than in this illustration, anormal eye will be depicted by a much larger pixel count to benoticeable. FIG. 3-e displays such a digitized eye. In this figure, aneye 350, as imaged on a sensor 60, will consist of roughly 25 pixelswide, 352. In particular interest for this invention the inner portion,the iris 360, in this case will be roughly 8 pixels in diameter, asillustrated in 462.

According to a preferred embodiment of this invention, the preview imageand the final image, or portions of them, need to be aligned as depictedin FIGS. 5 a to 5 c, block 540. As explained above, the reference imageand the final image may have different resolutions. The discrepancy inresolution may lead to differences in content, or pixel values, eventhough no data was changed in the subject image. In particular, edgeregions when downsampled and then upsampled may have a blurring or anaveraging effect on the pixels. Thus direct comparison of differentresolution images, even when aligned, may lead to false contouring. Inaddition, the reference image may be acquired prior to or after thefinal image is captured. Due to the above reasons, there is a need tomatch the two images, both in content and pixel resolution, as describedbelow.

FIG. 4 better illustrates the effect of the sub- and up-sample processin finding the difference pixelwise between two images. In this case,the input images are the ones illustrated in FIGS. 3-b and 3-drespectively high resolution and low resolution. In this figure, whitesquares such as 430 means that there is no difference between the two.Checkered squares or pixels, such as 420 means that there is adifference between the images.

The flat regions should display no significant differences due toresolution changes. The main difference will be caused be two reasonsnamely edge regions where changes in value occur such as in blocks 410.However, there is another cause for difference which is of interest tothis invention and displayed in 430. In these pixels, the difference iscaused by the actual change in the color of the eye from normal eyes tored-eyes. Not only is there a change in the pixel value but the changeis also more specifically reflected as change to a red or light colorfrom the normal color of the iris or form the black color of the pupil.

FIGS. 5-a to 5-d illustrate the workflow of the red-eye filter 90 ofthis embodiment, as well as variations thereof.

Referring first to FIG. 5-a, there are two input images into the filter,namely a full resolution image 510, I(x,y) which is the one that wascaptured by full depression of the shutter button and needs to beanalyzed for red-eye artefacts, and a preview image 520, P(x,y) which isused as a reference image and is nominally the same scene as the imageI(x,y) but taken without the flash. The preview image may be a result ofsome image processing taking into account multiple preview images andcreating a single image, 522. Methods of improving image quality basedon multiple images are familiar to those versed in the art of imageprocessing. The resulting output from the analysis process of 522 is asingle preview image.

The preview image 520 is normally, but not necessarily, of lowerresolution than the full resolution image 510, typically being generatedby clocking out a subset of the sensor cells or by averaging the rawsensor data. Therefore, the two images, or alternatively the relevantregions in the images (i.e. the regions containing or suspected tocontain eyes, which can be determined by image processing techniquesknown in the art), need to be matched in pixel resolution, 530. In thepresent context “pixel resolution” means the size of the image, orrelevant region, in terms of the number of pixels constituting the imageor region concerned. Such a process may be done by either upsampling thepreview image, 534, downsampling the acquired image, 532, or acombination thereof. Those familiar in the art will be aware of severaltechniques best used for such sampling methods. The result of step 530is a pair of images I′(x,y) and P′(x,y) corresponding to the originalimages I(x,y) and P(x,y), or relevant regions thereof, with matchingpixel resolution. The system and method of the preferred embodimentinvolves the detection and removal of red-eye artefacts. The actualremoval of the red-eye will eventually be performed on the fullresolution image. However, all or portions of the detection of red-eyecandidate pixel groupings, the subsequent testing of said pixelgroupings for determining false red-eye groupings, and the initial stepof the removal, where the image is presented to the user for userconfirmation of the correction, can be performed on the entire image,the subsampled image, or a subset of regions of the entire image or thesubsampled image.

Although nominally of the same scene, the preview image and the finallyacquired full resolution image may differ spatially due to the temporallag between capturing the two images. Therefore, the two images, orrelevant regions thereof, may need to be aligned, 540, especially inrespect of regions of the images containing or suspected to containeyes. Essentially, alignment means transforming at least one of theimages, and in this embodiment the preview image P′(x,y), to obtainmaximum correlation between the images, or relevant regions thereof,based on measurable characteristics such as color, texture, edgeanalysis. Those familiar in the art are aware of several algorithms toachieve such alignment; see, for example, U.S. Pat. No. 6,295,367 whichis hereby incorporated by reference and which describes alignment ofimages due to object and camera movement, and U.S. Pat. No. 5,933,546,which is also hereby incorporated by reference and which addresses theuse of multi-resolution data for pattern matching.

Further discussion on the alignment is presented in FIG. 5-c. In thisFigure, the inputs are the two images I′(x,y) and P′(x,y) as defined inFIG. 5-a. The alignment may be global for the entire image or local forspecific regions. For example, a simple linear alignment, such as ashift in the horizontal direction by H pixels, and/or in the verticaldirection by V pixels, or a combination of the two. Mathematically, theshifted image, P″(x,y), can be described as:P″(x,y)=P′(x−H,y−V)

However, simple translation operation may not suffice in the need toalign the image. Therefore, there may be a need for X-Y shearing, whichis a symmetrical shift of the object's points in the direction of theaxis to correct for perspective changes; X-Y tapering where the objectis pinched by shifting its coordinates towards the axis, the greater themagnitude of the coordinate the further the shift; or rotation around anarbitrary point.

In general, the alignment process may involve an affine transformation,defined as a special class of projective transformations that do notmove any objects from the affine space

 to the plane at infinity or conversely, or any transformation thatpreserves collinearity (i.e. all points lying on a line initially stilllie on a line after transformation) and ratios of distances (e.g., themidpoint of a line segment remains the midpoint after transformation).Geometric contraction, expansion, dilation, reflection, rotation, shear,similarity transformations, spiral similarities and translation are allaffine transformations, as are their combinations. In general, thealignment 540 may be achieved via an affine transformation which is acomposition of rotations, translations, dilations, and shears, allwell-known to one familiar in the art of image processing.

If it is determined through a correlation process that a globaltransformation suffices, as determined in block 542=YES, one of theimages, and for simplicity the preview image, will undergo an affinetransformation, 544, to align itself with the final full resolutionimage. Mathematically, this transformation can be depicted as:P″=AP′+qwhere A is a linear transformation and q is a translation.

However, in some cases a global transformation may not work well, inparticular for cases where the subject matter moved, as could happenwhen photographing animated objects. In such case, in particular inimages with multiple human subjects, and when the subjects move inindependent fashion, the process of alignment 540 may be broken down,546, to numerous local regions each with its own affine transformation.What is important is to align the eyes between the images. Therefore,according to this alternative, one or multiple local alignments may beperformed, 548, for regions in the vicinity surrounding the eyes, suchas faces.

Only after the images are aligned can one compare the potential red-eyecolors.

In the preferred embodiment of FIG. 5-a, the preview image informationis used as part of the falsing stage 96. Blocks 92, 94 and 98 correspondto the same blocks in FIG. 1, being the stages of pixel locator, shapeanalyzer and pixel modification respectively. This embodiment canincorporate pixel locator 92, shape analyzer 94 and pixel modifier 98 asdescribed in U.S. Pat. No. 6,407,777 (DeLuca), incorporated by referenceabove, the functions of the pixel locator 92 and shape analyzer 94 beingperformed on the image I′(x,y) and the pixel modifier 98 operating onthe original acquired image I(x,y). Block 96, which is the falsingstage, is improved in this embodiment as compared to the falsing stageof DeLuca.

Referring to block 96, for each region of the image I′(x,y) suspected asred-eye, step 596-2, as identified by steps 92 and 94, the suspectedregion is tested by comparing the pixel values of the region with thepixel values of the corresponding region in the aligned preview imageP″(x,y), 596-6. However, prior to doing so, the regions need to beprepared and modified for such comparison, 596-4.

Due to the fact that the regions may not match exactly, a pixel-by-pixelcomparison may not suffice. The reason for the mismatch may occur due tothe original size discrepancy. For example, in edges this phenomenon isgraphically illustrated in FIG. 4. Other reasons for a mismatch arepotential movement of the object, or there may be some averaging that isdone in the low resolution preview image that may loose high frequencycolor data. Such effects are referred to as smoothing and aliasing. Inaddition, even if the alignment is optimal, there may be sub-pixelalignment that can not be accounted for. Moreover, there may be colordifferences between the preview image, shot using available light andthe acquired full resolution image which is shot using flash. In manycases, the color transformation between one image to another is notglobal and uniform. Therefore, the process of preparing the regions forcomparison.

This process as illustrated in block 596-4 will be further described inFIG. 5-d. The underlying concept behind step 596-4 is to distinguishbetween differences that are caused due to the acquisition process andthe differences that are caused due to the existence of red-eye in theimage. This problem is well known to one familiar in the art ofstatistical pattern matching and scene analysis and image recognition.An example of such an application taking into account differences due toresolution is described in U.S. Pat. No. 5,933,546, which is herebyincorporated by reference.

If a region in the aligned preview image P″(x,y) was red and theequivalent region is red in the image I′(x,y), 596-6, that region willbe eliminated from I′(x,y) as a red-eye artefact, 596-9, and thecorresponding region will be eliminated as a red-eye artefact from theoriginal full resolution image I(x,y). Otherwise, the region willcontinue to remain suspected as red-eye, 596-8. The process willcontinue, 596-3, for all suspected regions.

The comparison of the regions for a color value is done as a globaloperation on the entire region, and the answer to the question ofwhether a region is red or not is made statistically for the entireregion and not pixel by pixel, i.e. it does not depend on the value ofany particular individual pixel. Such approach will account forinconsistencies on the pixel level that may be resolved statisticallywhen analyzing a larger collection of pixels consisting of a region. Forexample, some regions of the eye may not be fully red, or display otherartefacts such as a glint of high luminance. Other example for the needof a global statistical operation is the presence of noise in the image.Techniques are known in the art for such global comparison.

Based on the information above, the regions finally identified asred-eye artefacts can be modified, 98, to eliminate the red-eye from theoriginal full resolution image I(x,y). The modification can be doneusing any one of numerous available techniques such as luminancereduction, chrominance reduction, or subtraction of the artefact, asdescribed in US Published Patent Application 2002/0150306 (Baron), whichis hereby incorporated by reference.

FIG. 5-d describes the preparation of regions suspected of red-eye forcomparison as described in FIG. 5-a, block 596-4. As discussed above, asimple pixel level comparison may not be enough to determine whether theregion is not of red-eye nature. The process of preparation may includea combination of several components such as creating color balancebetween the regions of the preview image and the final image, 1510,analyzing the texture, or differences in high frequency patterns betweenthe two regions that may have occurred due to the change in resolution,1520, and comparing the edges between the two regions, 1530, where thedifferences may have occurred due to change in exposure, color balance,resolution or alignment, and in particular sub pixel alignment. Thecolor balance step 1510 comprises marking each red-eye region in I′(x,y)and the corresponding region in P″(x,y), steps 1512 and 1514,determining the difference in color balance between the region inI′(x,y) surrounding, but not including, the suspected red-eye region andthe corresponding region of P″(x,y), step 1516, and transforming theentire region, including the suspected red-eye region, based on thecolor balance difference so determined, step 1518.

As an alternative embodiment of this invention, the preview image can beused as part of the pixel locator stage 92, as illustrated in FIG. 5-b,rather than as part of the falsing analyzer 96. In FIG. 5-b, blocks 510,520, 522, 530, 532, 534, 540, 94 and 98 are identical to those in FIG.5-a. According to this embodiment, the use of the preview image in orderto detect red-eye artefacts is implemented as part of the red-eyeidentification process, otherwise described as the pixel locator 92 inFIG. 1 but here identified as Pixel Analyser and Region Segmenter 592.

After the suspected red-eye regions are identified, the processcontinues via the shape analysis 94, false detection elimination 96 andcorrection 98 as described in FIG. 1. In this case, the falsing detector96 may be performed according to DeLuca.

According to this embodiment, after the alignment step 540 the followingsteps 592-1 a and 592-1 b analyse both images I′(x,y) and P″(x,y) forthe presence of pixels having a color indicative of red-eye (592-1 a),for example in the manner of DeLuca, and then identifies clusters ofcontiguous red pixels so detected (592-1 b). This is known assegmentation and is more fully described in US Pat. Appn. 2002/0176623,which is hereby incorporated by reference.

Now, each region (cluster) with red content in the acquired imageI′(x,y), step 592-2, is compared with the corresponding region in thealigned preview image P″(x,y). The regions will need to be prepared,592-4, as previously described in relation to block 596-4 of FIG. 5-a.If the regions are red in both cases, 592-6=YES, the region will not bemarked as red-eye, no action will be taken and the process will continueto the next suspected region, 592-3. If the region is red in theacquired image I′(x,y) while the corresponding region is not red in thepreview image P″(x,y), 592-6=NO, then the region will be marked assuspected red-eye, 592-8.

FIG. 6-a shows a modification of the embodiment of FIG. 5-b in whichStep 540 (Align Images) has been divided into two steps, Step 541 (IfPossible Globally Align Images) and Step 592-3 (If Required LocallyAlign Images). Step 541 corresponds to Steps 542 and 544 of FIG. 5-c.However, if a global alignment is not possible or practical, the localalignment is deferred until after red pixel identification andclustering has been performed, since the presence of such clusters inthe two images I′(x,y) and P′(x,y) will assist in the local alignment.FIG. 6-b shows a similar modification applied to FIG. 5-a.

In the embodiments of the invention, in the comparison stages, 592-6 and596-6 the pixel values do not necessarily have to be compared with redbut may alternatively or additionally be compared with other values suchas yellow, white, pink, brown or other color indicative of a red-eyephenomenon, or to a range of values, to accommodate other flash relatedeye artefacts that are not purely red. Due to the fact that the eyesurface is retro-reflective (due to the smoothness created by the tears,and the spherical shape of the eyeball), the technique as described inthis specification can assist in the detection of the eyes in an image.Such existence of an eye can be found by comparison of the spectralreflection of the flash in the eye with the same region where no flashwas used, and thus without spectral reflection. This comparison mayassist in locating eyes in general and not just eyes with red-eyeartefacts. This process may be implemented by finding the change ofsmall specular reflections that occur in the eye region when flashillumination is used such as described in WO 03/026278 (Jarman). Thespecular reflections may be used as another indication of suspectedregions as defined in blocks 592-2 and 596-2 by comparing the specularreflection of the flash image with no specular reflection of the previewimage.

Alternatively to a binary decision of adding or eliminating a region,596-8 and 596-9, in the case of a continuous probability for eachregion, the process will be revised from a binary decision to changing aprobability decision. The quantitative determination of such change inprobability may be decided based on analysis of the confidence level ofthe comparison 592-4 and 596-4.

The preferred embodiments described above may be modified by adding orchanging operations, steps and/or components in many ways to produceadvantageous alternative embodiments. For example, the reference imagecan be a post-view image rather than a preview image, i.e. an imagetaken without flash immediately after the flash picture is taken.

A red-eye correction procedure may begin as described by block 92 withdetecting a human face in a digital image and, based on this detection,finding the eyes in the face (see, e.g., U.S. Pat. No. 6,252,976, U.S.Publ. Pat. App. No. 2003/0044070 and U.S. Pat. No. 6,278,491, which arehereby incorporated by reference). This procedure may also be used forcreating the regional alignment 546 and color balance 1510.

A range of alternative techniques may be employed to detect and verifythe existence of red-eye defects in an image (see, e.g., U.S. Publ. Pat.Apps. No. 2003/0044177 and 2003/0044178, which are hereby incorporatedby reference). These techniques may be incorporated into the pixellocator, shape analyzer, falsing analyzer and pixel modifiercorresponding to blocks 92, 94, 96 and 98. A camera may include softwareor firmware for automatically detecting a red-eye image using a varietyof image characteristics such as image brightness, contrast, thepresence of human skin and related colors. The analysis of these imagecharacteristics may be utilized, based on certain pre-determinedstatistical thresholds, to decide if red-eye defects exist and if aflash was used to take the original image.

The preferred embodiments described herein may involve expanded digitalacquisition technology that inherently involves digital cameras, butthat may be integrated with other devices such as cell-phones equippedwith an acquisition component, toy cameras etc. The digital camera orother image acquisition device of the preferred embodiment has thecapability to record not only image data, but also additional datareferred to as meta-data. The file header of an image file, such asJPEG, TIFF, JPEG-2000, etc., may include capture information includingthe preview image, for processing and red-eye detection at a later postprocessing stage, which may be performed in the acquisition device or ina separate device such as a personal computer. The preferred embodimentsdescribed herein serve to improve the detection of red-eyes in images,while eliminating or reducing the occurrence of false positives, and toimprove the correction of the detected artefacts.

While an exemplary drawing and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention, as set forth in the claims below and structural andfunctional equivalents thereof.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, unless expressly set forth orunderstood by those skilled in the art being necessary.

Thus, the preferred embodiments described herein provide an improvedmethod and apparatus for detecting red-eye phenomenon within imagestaken by a digital camera having a flash while eliminating or reducingthe occurrence of false positives by using preview information.

In addition to all of the references cited above that have beenincorporated by reference, the sections entitled BACKGROUND, SUMMARY OFTHE INVENTION, ABSTRACT and BRIEF DESCRIPTION OF THE DRAWINGS, arehereby incorporated by reference into the DETAILED DESCRIPTION OF THEPREFERRED EMBODIMENT.

1. A portable digital camera-enabled device having no photographic film,comprising: (a) an integral flash for providing illumination duringimage acquisition; (b) a digital image capturing apparatus including (i)at least one lens; (ii) a main image sensor for acquiring a main imageof a scene including at least one real face (contra an image of a face),and (iii) a separate sensor for acquiring a reference image including arepresentation nominally of said same scene as the main image includingsaid at least one real face; (c) a face filter for correcting a facedefect within the main image acquired by the main image sensor based onanalysis of the reference image acquired by the separate sensorincluding said representation nominally of said same scene as the mainimage including said at least one real face; (c) a dedicated hardwaresubsystem which is provided reference image data from the separatesensor; and (d) a main system memory which receives raw image data andmetadata as implemented by the dedicated hardware subsystem; and (f)wherein the main system memory is configured to receive meta data thatcomprises identified defect areas determined within at least one imageof said scene including at least one real face.
 2. The portable digitalcamera-enabled device of claim 1, wherein the reference image comprisesa low resolution image including a subsample representation.
 3. Theportable digital camera-enabled device of claim 2, wherein the referenceimage comprises a subsample resolution representation.
 4. The portabledigital camera-enabled device of claim 2, wherein the reference imagecomprises a segmented representation of the main image.
 5. The portabledigital camera-enabled device of claim 2, further comprising a modulefor changing a degree of subsampling.
 6. The portable digitalcamera-enabled device of claim 1, wherein the reference image comprisesa precapture image captured just before the main image is captured. 7.The portable digital camera-enabled device of claim 1, furthercomprising one or more further filters for correcting exposure, focus orcomposition of the main image, or combinations thereof, based onanalysis of the reference image acquired by the separate sensor.
 8. Theportable digital camera-enabled device of claim 1, further comprising afilter for aligning the reference image with the main image.
 9. Theportable digital camera-enabled device of claim 8, wherein the aligningcomprises estimating camera or object movement, or both, in one or moreintervals between capture of the main and reference images.
 10. Theportable digital camera-enabled device of claim 1, wherein the referenceis generated in parallel with the main image.
 11. A method of correctinga face defect within a digital image, comprising: providing a portabledigital camera-enabled device including a main image sensor forcapturing a main image and a separate sensor for acquiring a referenceimage temporally proximate to acquisition of the main image; providingillumination during main image acquisition with a flash; acquiring withthe main image sensor a main image of a scene including at least onereal face (contra an image of a face); acquiring with the separatesensor a reference image temporally proximate to acquisition of the mainimage and including a representation nominally of said same scene as themain image including said at least one real face; correcting a facedefect within the main image acquired by the main image sensor based onan analysis of the reference image acquired by the separate sensorincluding the representation nominally of said same scene as the mainimage including said at least one real face; providing reference imagedata to a dedicated hardware subsystem from the separate sensor; andproviding raw image data and meta data to a main system memory, the metadata comprising identified defect areas determined within at least oneimage of said scene including at least one real face.
 12. The method ofclaim 11, wherein the reference image comprises a low resolution imageincluding a subsample representation.
 13. The method of claim 12,wherein the reference image comprises a subsample resolutionrepresentation.
 14. The method of claim 12, wherein the reference imagecomprises a segmented representation of the main image.
 15. The methodof claim 12, further comprising changing a degree of subsampling. 16.The method of claim 11, wherein the reference image comprises aprecapture image captured just before the main image is captured. 17.The method of claim 11, further comprising correcting exposure, focus orcomposition of the main image, or combinations thereof, based onanalysis of the reference image acquired by the separate sensor.
 18. Themethod of claim 11, further comprising aligning the reference image withthe main image.
 19. The method of claim 18, wherein the aligningcomprises estimating camera or object movement, or both, in one or moreintervals between capture of the main and reference images.
 20. Themethod of claim 11, further comprising generating the reference image inparallel with the main image.
 21. One or more processor-readable digitalstorage media having code embodied therein for programming one or moreprocessors of a portable digital camera-enabled device that includes amain image sensor and a separate reference image sensor to perform amethod of correcting a face defect within a digital image, wherein themethod comprises: providing illumination during main image acquisitionwith a flash; acquiring with the main image sensor a main image of ascene including at least one real face (contra an image of a face);acquiring with the separate sensor a reference image temporallyproximate to acquisition of the main image and including arepresentation nominally of said same scene as the main image includingsaid at least one real face; correcting a face defect within the mainimage acquired by the main image sensor based on an analysis of thereference image acquired by the separate sensor including therepresentation nominally of said same scene as the main image includingsaid at least one real face; providing reference image data to adedicated hardware subsystem from the separate sensor; and providing rawimage data and meta data to a main system memory, the meta datacomprising identified defect areas determined within at least one imageof said scene including at least one real face.
 22. The one or moreprocessor-readable digital storage media of claim 21, wherein thereference image comprises a low resolution image including a subsamplerepresentation.
 23. The one or more processor-readable digital storagemedia of claim 22, wherein the reference image comprises a subsampleresolution representation.
 24. The one or more processor-readabledigital storage media of claim 22, wherein the reference image comprisesa segmented representation of the main image.
 25. The one or moreprocessor-readable digital storage media of claim 22, wherein the methodfurther comprises changing a degree of subsampling.
 26. The one or moreprocessor-readable digital storage media of claim 21, wherein thereference image comprises a precapture image captured just before themain image is captured.
 27. The one or more processor-readable digitalstorage media of claim 21, wherein the method further comprisescorrecting exposure, focus or composition of the main image, orcombinations thereof, based on analysis of the reference image acquiredby the separate sensor.
 28. The one or more processor-readable digitalstorage media of claim 21, wherein the method further comprises aligningthe reference image with the main image.
 29. The one or moreprocessor-readable digital storage media of claim 28, wherein thealigning comprises estimating camera or object movement, or both, in oneor more intervals between capture of the main and reference images. 30.The one or more processor-readable digital storage media of claim 21,wherein the method further comprises generating the reference image inparallel with the main image.